Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

improve gym env / tune hps #71

Merged
merged 33 commits into from
Feb 1, 2024
Merged

improve gym env / tune hps #71

merged 33 commits into from
Feb 1, 2024

Conversation

Armandpl
Copy link
Owner

@Armandpl Armandpl commented Feb 1, 2024 •
edited
Loading

  • setup new reward function and sweep over them
    • nothing very conclusive, each of those seem to converge in about the same time and lead to a working policy
      • slight doubt about the cos reward because one of the run converged to a bad policy. todo (later): run more runs comparing the cos reward to another one and look at the resulting policies.
    • the idea was to use the exponential to have higher rewards when above pi/2 to try and get the agent to converge faster. It seems to be the case but the exp makes the reward more flat close to 0 which seems to make the early exploration slower.
  • setup tiny sweep over tqc hps:
    • tiny sweep because each run takes ~10min so we can't sweep over that much hyper-parameters
    • using sbx and jax could speed it up but I couldn't nicely setup jax cuda with poetry
    • found that a higher learning rate converges faster, other hp didn't seem to have an effect. todo later: use rliable to properly eval the result of sweeps!
  • add raw angles to the obs when limits depend on them, so as to not break the markov assumption
  • add a pid to slow down the motor on reset, trying to gain some wall time during training. this is currently badly tuned and may have damaged the motor. need to investigate further
  • delete unused code: scripts/train_sac.pyand scripts/robot_inference.py
  • add DeadZone wrapper, ideally we'll get rid of it once we figure out why the agent doesn't explore properly without it
  • debug issue where it seemed gsde wasn't used with tqc on sbx
    • we were updating the policy at each step which made it look like gaussian noise
  • setup training on mac
  • fix bug in FurutaReal env reset: was only working between -pi and pi, didn't take into account the pendulum could do multiple turn. it meant we were waiting until reset timeout when we didn't need to

@Armandpl
use pid to speed up real env reset
9086aed
@Armandpl
setup deadzone wrapper
01451f4
@Armandpl
setup tqdc + deadzone sim exp
9e7d5fa
@Armandpl
add angles to the obs when limiting them; add time feat wrapper
347befd
@Armandpl
check for nan instead of none bc none gets converted to nan by np.arr…
cd9cf1c 
…ay w/ dtype float32
@Armandpl
setup real exp
5c4b6c4
@Armandpl
add deadzone wrapper
e2d6360
@Armandpl
use cos for env reset bc pendulum can do multiple turns
99a80dd
@Armandpl
add a center to the deadzone wrapper to allow for a zero action
df6cf31
@Armandpl
add exp reward
b5bdca6
@Armandpl
setup tqc + exp reward sim experiment
3e00fae
@Armandpl
add sbx and jax-metal
06116f7
@Armandpl
fix base wrapper; make sim exp closer to real w/ n_envs=1
7c9f844
@Armandpl
add timeout to motor stopping, hack to avoid pid oscillations instead…
22c5a43 
… of tuning it
@Armandpl
setup rew sweep
8a62cdc
@Armandpl
investigate tqc action looking like gaussian noise
1b925e6
@Armandpl
fix default args
d650c03
@Armandpl
add max act to deadzone wrapper
0f6c7ad
@Armandpl
setup hp tune sweep for tqc
5503027
@Armandpl
actually use early stopping
5b7e11d
@Armandpl
use sb3_contrib master to train
a46febd
@Armandpl
use progressbar
e3d3761
@Armandpl
add option to use either sbx or sbx tqc
2186f79
@Armandpl
add new reward taking theta into account
df26804
@Armandpl
remove unused code
f679592
@Armandpl
make progressbar a param
722456e
@Armandpl
setup new rew sweep
e751650
@Armandpl
fix error in reward
2d76294
@Armandpl
setup real training
5c933d0
@Armandpl
fix sweep setup
9759dc8
@Armandpl Armandpl linked an issue Feb 1, 2024 that may be closed by this pull request
sanity check by training with sac or tqc #64
Open
@Armandpl Armandpl changed the title improve gym env / tune hp improve gym env / tune hps Feb 1, 2024
@Armandpl Armandpl merged commit f12a1da into master Feb 1, 2024
2 checks passed
@Armandpl Armandpl deleted the sanity-check-robot branch February 1, 2024 16:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

sanity check by training with sac or tqc
1 participant
@Armandpl